Changes on the Horizon for the Multimedia Community

The Impact of Deep Learning The development of AI algorithms, represented by deep learning, has bolstered multimedia research. In particular, deep learning has led to a multimodality-based algorithm framework, enabling the effective fusion and use of cross-domain data. Take image and video captioning, for example. A couple of years ago, tagging was the only way to describe images and videos. But now, deep learning has helped establish the connection between computer vision and natural-language processing (NLP), resulting in tagging being replaced by coherent, naturallanguage sentence generation. Image captioning took inspiration from recent advances in machine translation. The captioning process is based on a Convolutional Neural Network (CNN) that encodes an image into a compact representation. Then, a Recurrent Neural Network (RNN)/Long Short Term Memory (LSTM) generates the corresponding sentence. The latest research has also introduced interdisciplinary theories and methodology into video captioning. Unlike image captioning, both spatial details in a single frame and temporal information between frames play an important role in video captioning. Consequently, it is usually more complex than image captioning. As related fields and hardware continue to develop, the generation of multiple sentences or even entire paragraphs could become a reality for image and video captioning. Moreover, more natural user interaction systems could become available, and modalities might not be limited to computer vision and NLP. Instead, voice, depth, and text features could be further incorporated into the deep learning pipeline. However, despite research progress in academia, lots of problems still stand in the way of applying deep learning in real life. For example, interpretability is crucial for multimodality applications in the medical domain, so we need to delve deeper in this area. Also, in mobile applications, imageand videocaptioning tasks require a tradeoff between speed and accuracy due to the graph search required for inferencing. We’ve still got a long way to go, but looking to the future, as deep learning bridges multiple modalities and we acquire more information, it should provide better AI services to users.